Skip to content

FastPLMs 1.0 repository and compliance upgrade#39

Open
lhallee wants to merge 19 commits into
mainfrom
clean_up
Open

FastPLMs 1.0 repository and compliance upgrade#39
lhallee wants to merge 19 commits into
mainfrom
clean_up

Conversation

@lhallee

@lhallee lhallee commented Jul 15, 2026

Copy link
Copy Markdown
Contributor 
## Summary

This PR upgrades FastPLMs to the 1.0 repository, packaging, compliance, and runtime architecture. It replaces the flat source and Docker layout with a `src/fastplms` package, typed model manifest, isolated upstream parity oracles, consolidated containers, manifest-driven artifacts, structured test tiers, shared attention and embedding APIs, and substantially rewritten documentation.

This is intentionally a clean API break. Production code does not import official repositories, runtime images do not contain upstream sources or checkpoint weights, and checkpoint compatibility is preserved through stable parameter keys or deterministic state transforms.

## Repository and packaging

- Move production code under `src/fastplms/`.
- Split attention, embeddings, models, registry, and runtime responsibilities.
- Add `pyproject.toml` and `uv.lock` for FastPLMs 1.0.
- Set Python support to `>=3.11,<3.15`.
- Lock GPU validation to Python 3.12, PyTorch 2.13.0, Transformers 5.13.0, and CUDA 13.
- Keep core dependencies limited to the runtime model stack.
- Separate `structure`, `flash`, `fp8`, `train`, and `dev` extras.
- Remove import-time downloads, compilation, tokenizer construction, and global backend mutations.
- Reorganize examples, benchmarks, containers, tools, model cards, licenses, tests, and documentation into dedicated directories.
- Remove the superseded Dockerfile fleet, flat package tree, legacy test layout, and one-off upload/build scripts.

## Typed model manifest and provenance

- Add `src/fastplms/models.toml` as the authoritative declaration for:
  - checkpoint identifiers and immutable revisions;
  - file identities and safetensors hashes;
  - AutoClasses and tokenizer behavior;
  - deterministic state transforms;
  - supported attention and precision paths;
  - native reference containers;
  - test tiers;
  - checkpoint and source licenses;
  - conversion provenance;
  - documentation state.
- Generate the support matrix and model cards from the manifest.
- Add exact state, dtype, shape, alias, tokenizer, and semantic configuration contracts.
- Move every official repository under `vendor/upstream/` as a real pinned Git submodule:
  - ANKH
  - Biohub ESM
  - Biohub Transformers
  - Boltz
  - DPLM
  - E1
  - fair-esm
  - OpenFold
  - ProteinTTT
- Document that upstream modules are isolated parity oracles rather than production dependencies.
- Add project-wide third-party notices, license records, attribution, provenance, and modified-file notices.

## Attention and kernel handling

- Add a shared attention layer with centralized 2D masks, 4D masks, and Flex `BlockMask` construction.
- Support explicit eager, SDPA, Flex Attention, FlashAttention 2, and FlashAttention 3 declarations where supported by each model family.
- Reject unavailable or unsupported requested backends instead of silently falling back.
- Add compiled Flex mask and function caching keyed by device, dtype, shape, sequence lengths, and mask semantics.
- Use only revision-pinned, precompiled Hugging Face `kernels` FlashAttention binaries.
- Do not install or compile the `flash-attn` source package.
- Add `kernels.lock` and release checks for immutable kernel revisions and compatible binary variants.

## Shared embedding API

- Add `fastplms.embed_dataset(...)` and `model.embed_dataset(...)`.
- Add ordered `EmbeddingInput`, `EmbeddingRecord`, and `EmbeddingResult` types.
- Preserve input order, FASTA identifiers, and duplicate records.
- Make dictionary conversion reject duplicate identifiers unless the caller explicitly selects another policy.
- Pool biological residues only, excluding BOS, EOS, padding, chain delimiters, and non-protein structure tokens.
- Support mean, max, norm, median, standard deviation, variance, CLS, and PARTI where the model has the required semantics.
- Replace the NetworkX PageRank dependency with Torch power iteration.
- Add ordered ragged full-residue embeddings.
- Add 2 GiB-sharded safetensors with JSON indexing and run manifests.
- Add transactional SQLite streaming and exact resume.
- Persist checkpoint identity, hashes, dtype, backend, layer or projection, token policy, truncation, pooling, package versions, and input fingerprint.
- Retain read-only legacy `.pth` import behind explicit unsafe-pickle opt-in.

## ESM++ and ESMC

- Preserve the official ESMC:
  - QKV projection and ordering;
  - full-width Q/K normalization;
  - RoPE placement;
  - attention scale;
  - SwiGLU feed-forward structure;
  - residual scaling;
  - final normalization;
  - language-model head;
  - parameter and tokenizer contracts.
- Match the Biohub Transformers hidden-state convention: the input to every transformer block followed by the final normalized state.
- Preserve the 81-state ESMC-6B ordering consumed by ESMFold2.
- Infer a missing padding mask from `input_ids != pad_token_id`.
- Match Biohub equality-based `sequence_id` semantics for biological and padding groups.
- Recompute RoPE frequencies on the execution device, matching the pinned Biohub Transformers CUDA policy.
- Add full-tensor SDPA exactness gates covering biological residues, special tokens, and padding.
- Keep backend-specific BF16 numerical observations in the ESM++ model cards rather than the main README.
- The observed alternate-backend errors came from one seeded batch of randomly generated compliance sequences and are not presented as expected error distributions for biological sequences.
- The declared `AutoModel` and `AutoModelForMaskedLM` surfaces are the official-equivalent ESM++ interfaces.

## ESMFold2 learned representation

- Support exactly four ESMFold2 variants:
  - `biohub/ESMFold2`
  - `biohub/ESMFold2-Fast`
  - `biohub/ESMFold2-Experimental-Cutoff2025`
  - `biohub/ESMFold2-Experimental-Fast-Cutoff2025`
- Add:

  ```python
  model.project_esmc_hidden_states(H)

where H: (b, l, 81, 2560) produces Z: (b, l, 256).

  • Preserve base_z_linear, base_z_combine, and checkpoint parameter keys.
  • Return the learned sequence representation before pair features are created.
  • Expose the learned projection through the shared dataset embedding API.
  • Restrict ESMFold2 dataset embeddings to single-chain residue sequences.
  • Support residue-statistic pooling while rejecting CLS, PARTI, complexes, ligands, MSAs, and chain-separated embedding inputs.

ESMFold2 FP8

  • Add esmc_precision="auto" | "bf16" | "fp32" | "fp8".
  • Add reload_esmc(...) and structured precision status.
  • Select FP8 automatically only for a direct load onto a supported CUDA device when Transformer Engine reports availability.
  • Keep explicit FP8 strict.
  • Preserve BF16 for gradient-enabled and test-time-training paths.
  • Retain canonical BF16 weights and rebuild transient Transformer Engine state at startup or reload.
  • Never serialize runtime-quantized state.
  • Convert only the 80 ESMC attention output projections in the validated FP8 path.
  • Cover odd token lengths, mixed padding, finite outputs, repeated reloads, projection parity, and seeded folding metrics.

Other models and feature paths

  • Retain ESM2, ESM++, ESM3, E1, DPLM, DPLM2, ANKH, ESMFold, ESMFold2, Boltz2, TTT, binder-design, and fine-tuning functionality under the new package.
  • Restore official DPLM2 tied-language-model and contact-head behavior.
  • Preserve E1 tokenizer-free raw-sequence preparation and RAG adapters.
  • Match ANKH official encoder and sequence-to-sequence heads.
  • Keep the synthesized ANKH masked-LM path explicitly identified as a FastPLMs extension.
  • Add structured coverage for generation, multimodal inputs, TTT, binder workflows, conversion, and source independence.
  • Mark Boltz2 provisional. It remains available, but unfinished native BF16 end-to-end equivalence does not block the ESM++ or ESMFold2 release gates.

Containers and remote validation

  • Replace the Dockerfile fleet with one multi-stage Dockerfile, Buildx Bake targets, and Compose services.
  • Keep candidate and runtime images separate from native reference images.
  • Give each reference image only its corresponding upstream checkout and legal material.
  • Add a portable SSH runner accepting workstation and identity parameters at runtime.
  • Do not record workstation details, key paths, or secrets in tracked files.
  • Preserve model and build caches.
  • Return JUnit, JSON, structure, and benchmark outputs from remote runs.
  • Centralize GPU access, ipc: host, cache volumes, source mounting, and output directories.

Local Hub artifacts

  • Add a deterministic manifest-driven artifact builder.
  • Write explicit safetensors shards, AutoClass metadata, tokenizer assets, model cards, licenses, notices, and provenance.
  • Validate fresh offline remote-code loading with FastPLMs absent from sys.path.
  • Exercise every advertised AutoClass.
  • Run inference, save/reload, and compare with package source.
  • Keep artifact builds and Hub publication separate.
  • This work does not upload, delete, or modify live Hub repositories.

Testing and compliance

  • Replace the legacy test layout with:
    • tests/unit
    • tests/integration
    • tests/parity
    • tests/structure
    • tests/release
  • Separate routine checks, live compliance, structure, feature, artifact, and benchmark suites.
  • Treat missing expected dependencies and advertised backends as failures rather than skips.
  • Add exact configuration, tokenizer, state, alias, inference, embedding, structure, source-independence, licensing, and artifact contracts.
  • Keep official repositories isolated in native reference containers.
  • Use safetensors-native reference exchanges rather than importing upstream packages into candidate runtime images.
  • Add normalized relative L2, relative Q99.9, residue cosine, pooled cosine, confident top-1 agreement, and probability JSD metrics.
  • Add strict folding metrics for prepared features, diffusion noise, geometry, RMSD, lDDT-Cα, pLDDT, PAE, pTM, ipTM, and probability divergence.
  • Keep benchmark execution outside pytest, with only a short smoke test inside pytest.

Benchmarking

  • Add a standalone H100 benchmark suite using pre-tokenized, preallocated GPU inputs.
  • Report:
    • model loading;
    • first forward;
    • compilation;
    • steady-state forward;
    • full embedding;
    • ESMFold2 projection;
    • BF16 and FP8 execution.
  • Measure latency, logical and padded throughput, padding efficiency, P95 latency, peak memory, compile time, first-forward time, temperatures, clocks, and exact environment revisions.
  • Store raw samples.
  • Add statistical throughput and memory regression rules.
  • Never update performance baselines automatically.
  • Make performance claims only for the validated H100 environment.

Documentation

  • Rewrite the README as a concise installation and usage entry point.
  • Add or rewrite:
    • architecture;
    • model support;
    • embeddings;
    • attention;
    • ESMFold2;
    • testing;
    • benchmarking;
    • artifacts;
    • migration;
    • licensing;
    • contribution;
    • TTT;
    • binder-design;
    • fine-tuning documentation.
  • Generate support tables and checkpoint cards from the manifest.
  • Enforce lowercase scalar and dimension notation, uppercase mathematical tensor aliases, and lowercase parenthesized shapes.
  • Add executable snippet, link, notation, claim, and generated-document checks.
  • Keep detailed ESM++ alternate-kernel reproducibility context in model cards rather than advertising it in the README.

Validation environment

All verification was run on the supplied H100 workstation using:

  • Python 3.12.3
  • PyTorch 2.13.0+cu130
  • Transformers 5.13.0
  • CUDA 13.0/13.0.1 container runtime
  • Hugging Face kernels 0.15.2
  • No source-installed flash-attn

Validation results

ESM++ and ESMC

  • 19 masking, RoPE, source-independence, and focused native-parity tests passed.
  • 3 representative full-tensor FP32/BF16 SDPA exactness gates passed.
  • 2 additional full-tensor BF16 gates passed for ESMC-600M and ESMC-6B.
  • 3 exact configuration, state, alias, tokenizer, and tokenizer-asset contracts passed across ESMC-300M, ESMC-600M, and ESMC-6B.
  • Default SDPA is bit-for-bit equal to the pinned official implementation at every hidden state and across residues, BOS/EOS, and padding.
  • The parity metric selects biological residues before comparison. Alternate-backend observations are therefore neither padding-contaminated nor MSE.
  • The canonical compliance route uses isolated native-reference results. Candidate images intentionally cannot import the Biohub Transformers oracle.

ESMFold2

  • 15/15 focused ESMFold2 release tests passed across all four supported variants.
  • Coverage includes:
    • learned projection parity;
    • BF16 folding;
    • FP8 folding;
    • automatic and explicit precision policy;
    • strict unavailable-device behavior;
    • odd lengths;
    • mixed padding;
    • finite outputs;
    • three BF16-to-FP8 reload cycles;
    • structure metrics.
  • The four-variant ESMFold2 gate passed on the H100 validation stack.

Artifacts and release contracts

  • 17/17 offline artifact-loading and save/reload tests passed for the generated validation artifacts.
  • 110/110 static shipping and release-contract checks passed in the completed shipping slice.
  • The final documentation, registry, model-card, and license validation run passed 75/75 tests.
  • A subsequent focused model-card regeneration and consistency run passed 6/6 tests.
  • Manifest-derived document generation passes --check.
  • git diff --check passes.

Known boundaries and release handoff

  • Boltz2 remains provisional and is not part of the ESM++ or ESMFold2 release gate.
  • Generated dist/hub artifacts are build outputs and are not committed.
  • The final ESM++ masking and RoPE audit changed tracked runtime source after the earlier offline artifact run.
  • Release artifacts should therefore be regenerated from this commit and the offline artifact suite rerun before publishing or tagging.
  • Alternate ESM++ BF16 kernels produced relative L2 results of approximately 0.01 to 0.012 on one seeded random compliance batch. This is documented as reproducibility context rather than a general biological-sequence error estimate.
  • No live Hub repositories were uploaded, modified, or deleted.
  • No release tag was created.

Review guidance

The highest-value review surfaces are:

  1. src/fastplms/models.toml and src/fastplms/registry.py for the authoritative support and provenance contract.
  2. src/fastplms/attention/ and kernels.lock for backend selection, masking, and precompiled FlashAttention policy.
  3. src/fastplms/embeddings/ for residue selection, pooling, persistence, and resume behavior.
  4. src/fastplms/models/esm_plusplus/ for ESMC architecture and exact SDPA behavior.
  5. src/fastplms/models/esmfold2/ for the learned representation and precision/reload policy.
  6. tests/parity/, tests/structure/, and tests/release/ for the compliance and release gates.
  7. docker/, tools/artifacts/, and tools/remote/ for reproducible validation and distribution.

@lhallee

lhallee commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

#39 is not merge-ready. The custom Transformers v5 loading path is healthy, but exact-head testing on the proposed dependency stack found model-correctness failures and broken release gates.

Blocking findings

  1. [P1] ESMFold2 experimental FP8 fails its hard structure threshold.

    Full isolated four-variant compliance produced 14/15 passes. esmfold2_experimental_cutoff2025 scored 0.9089 lDDT-Cα, below the hard minimum of 0.95 enforced by the [ESMFold2 parity test](

    FastPLMs/tests/structure/test_esmfold2.py

    Line 541 in 7f182e0

    with common_module._seed_context(0), torch.no_grad():
    ). The other variants and all fresh-reload tests passed.

  2. [P1] ESMC exact checkpoint and numerical gates are red.

    Regenerating native references and artifacts from the PR head yielded 7 passed, 7 failed:

    • Small, large, and 6B exact-checkpoint contracts fail with KeyError: 'classifier_dropout'.
    • BF16 ESM++ misses the documented relative_l2 <= 0.01 engineering target for eager, FlexAttention, FlashAttention 2, and FlashAttention 3, ranging from 0.01001 to 0.01040.
    • FP32 eager/SDPA/Flex, BF16 SDPA, and all three checkpoint inference cases passed.

    These are failures in the PR’s own [native-results suite](

    FastPLMs/tests/parity/test_native_results.py

    Line 105 in 7f182e0

    fast = _load_fast(spec, torch.device("cpu"), None)
    ), not stale references.

  3. [P1] Locked offline FlashAttention artifacts do not work under the proposed defaults.

    Even after the repository-prescribed kernel download, both locked Flash artifact tests fail. huggingface_hub==1.23.0 reports an IncompleteSnapshotError, with 196 files missing from the cached kernel snapshot, and offline loading then aborts. This breaks the intended published/offline path covered by [test_published_automodel.py](https://github.com/Synthyra/FastPLMs/blob/7f182e053bc45cf867e28bc6e07d9a6f290c57eb/tests/release/test_published_automodel.py#L344).

  4. [P1] Embedding resume can silently reuse results from different weights.

    The resume fingerprint in [runner.py](https://github.com/Synthyra/FastPLMs/blob/7f182e053bc45cf867e28bc6e07d9a6f290c57eb/src/fastplms/embeddings/runner.py#L341) identifies configuration/checkpoint metadata, but not the current in-memory weights. A fine-tuned or mutated model with the same identity can therefore return stale “complete” embeddings. That is a scientific provenance and correctness issue.

  5. [P1] The documented artifact builder can package the wrong tokenizer with false provenance.

    [tools/artifacts/build.py](https://github.com/Synthyra/FastPLMs/blob/7f182e053bc45cf867e28bc6e07d9a6f290c57eb/tools/artifacts/build.py#L933) records the official tokenizer provenance, but its documented direct CLI lacks a tokenizer-directory option and may copy the selected checkpoint tokenizer instead. The fast and official tokenizer configurations have different recorded hashes.

Additional actionable problems

Positive result

On the requested canonical environment:

  • Python 3.12.3
  • PyTorch 2.13.0+cu130
  • Transformers 5.13.0
  • huggingface_hub 1.23.0
  • kernels 0.15.2
  • H100 80 GB

Offline trust_remote_code=True loading, inference, and save/reload passed:

  • ESMC: 9/9 across small, large, and 6B using AutoConfig, AutoModel, and AutoModelForMaskedLM
  • ESMFold2: 8/8 across all four variants using AutoConfig and AutoModel

So the earlier Transformers v5 custom-class/weight-loading concern appears resolved. It does not offset the ESMC parity, ESMFold2 FP8, offline kernel, and provenance blockers above.

Recommendation: request changes. At minimum, fix or remove the failing experimental ESMFold2 FP8 artifact, restore all ESMC contract/engineering gates, repair locked offline kernel loading on the proposed versions, and correct the embedding/artifact provenance issues before merging.

@lhallee

lhallee commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

PR 39 merge-readiness implementation and validation report

Recommendation

This work materially improves PR 39, and the ESMFold2 precision policy is now
scientifically defensible: auto always resolves to BF16, while FP8 remains an
explicit, inference-only opt-in. I do not recommend merging yet. The remaining
blockers are the failed ESMC BF16 backend target, completion of the aggregate
exact-head release suite, and an approving review.

Validation was performed from a working tree based on PR head
7f182e053bc45cf867e28bc6e07d9a6f290c57eb. The changes described below are
not committed, so this is not yet an exact committed-head release report.

What changed

Model correctness and Transformers 5 compatibility

  • Replaced the duplicated official and candidate semantic-config extraction
    logic with one helper. The comparison now includes classifier_dropout,
    initializer_range, and tie_word_embeddings.
  • Corrected the ESM++ base AutoModel contract so it owns the checkpoint's
    unused sequence_head parameters. This eliminates unexpected keys without
    changing base-model forward semantics and preserves exact loading through
    AutoModelForMaskedLM.
  • Retained all advertised ESMC attention backends: eager, SDPA,
    FlexAttention, FlashAttention 2, and FlashAttention 3.
  • Added Transformers 5.13 load, save, fresh-interpreter reload, offline, and
    sharded-checkpoint coverage for ESM++ small, large, and 6B and all four
    ESMFold2 variants.

ESMFold2 policy and biological validation

  • Changed esmc_precision="auto" to resolve deterministically to BF16 on every
    device, including H100 systems with Transformer Engine installed.
  • Preserved explicit esmc_precision="fp8" and
    model.reload_esmc(precision="fp8", device="cuda"). Explicit FP8 remains
    fail-closed when the hardware or Transformer Engine path is unavailable.
  • Documented the explicit FP8 option, exact runtime behavior, real-protein test
    design, per-variant pass counts, and numerical discrepancy ranges in the root
    README, ESMFold2 guide, testing and migration guides, and all four ESMFold2
    model cards.
  • Added a pinned fixture manifest for five complete, experimentally determined
    chains: Crambin (1CRN:A), protein G B1 (1PGA:A), bovine pancreatic
    trypsin inhibitor (5PTI:A), ubiquitin (1UBQ:A), and GFP (1EMA:A). The
    manifest records source hashes, exact sequences, lengths, curated-record
    cross-checks, and polymer-to-observed-residue mappings.
  • Reworked isolated official comparisons so every variant uses official BF16,
    FastPLMs BF16, and FastPLMs FP8 with identical features, masks, diffusion
    noise, and schedules at seeds 17, 29, and 43.
  • Recorded every structure, confidence, probability, experimental-coordinate,
    geometry, and per-residue metric before assertions. Failing and near-threshold
    cases persist superposed structures and per-residue diagnostic tables.
  • No random sequences, generated sequences, or length-selected fragments are
    used for ESMFold2 structure validation.

Runtime and shared APIs

  • Added embedding run-fingerprint schema v2: an exact SHA-256 over sorted model
    parameters and persistent buffers, streamed through bounded CPU buffers.
  • Added keyword-only model_state_fingerprint for offloaded, custom, or
    caller-owned model identity. Resume refuses older fingerprint schemas.
  • Restored store_all_hidden_states=True with the required
    full_embeddings=True contract, (batch, states, tokens, width) internal
    handling, per-sequence (states, biological residues, width) outputs, and
    safetensors and SQLite round trips.
  • Routed causal FlashAttention with a 2D padding mask through the varlen path,
    propagated causal=True, validated mask shape, and zeroed padded query rows.
  • Made DPLM2 contact prediction infer padding when attention_mask=None, while
    retaining explicit-mask authority.

Artifacts, offline kernels, packaging, and release automation

  • Made offline FlashAttention kernel resolution accept only the exact locked
    repository, revision, compatible cached variant, and digest. Missing,
    ambiguous, corrupt, and revision-mismatched caches now fail closed.
  • Added offline FA2 and FA3 artifact inference after kernel prefetch.
  • Added --tokenizer-dir to the direct artifact builder and pinned FastPLMs
    artifacts to the official tokenizer snapshot actually copied into the
    artifact.
  • Isolated official ESMFold2 reference bundles from candidate images. Candidate
    images no longer install Biohub ESMFold2 or zstd for comparisons.
  • Split static validation from H100 and Transformer Engine validation and
    expanded the aggregate remote release orchestration.
  • Synchronized Compose named build contexts with Bake and added parity tests.
  • Packaged THIRD_PARTY_NOTICES.md under wheel license metadata and verified
    its exact bytes.
  • Removed patch whitespace while preserving the ANKH third-party license bytes
    through a targeted whitespace exception.
  • Added a non-required hosted CPU workflow for lint, static checks, CPU tests,
    wheel legality, Compose/Bake parity, and diff checking.
  • Removed unverified blanket claims that all tests pass.

Locked H100 validation environment

Component Version
Python 3.12.3
PyTorch 2.13.0+cu130
Transformers 5.13.0
huggingface_hub 1.23.0
kernels 0.15.2
Transformer Engine 2.12.0
CUDA 13.0.1
GPU NVIDIA H100 PCIe 80 GB

Test results

Focused and CPU validation

Check Result
Initial integrated focused suite 143 passed
Final focused regression selection 140 passed, 9 deselected by marker selection
Full CPU structure-image suite 414 passed, 3 failed
Final ESMFold2 auto/explicit precision policy selection 6 passed, 46 deselected
H100 auto reload and embedding execution 1 passed
Python compilation passed
Ruff fatal-error selection (E9,F63,F7,F82) passed
Combined working-tree git diff --check origin/main passed

The three full-CPU failures are known artifact-golden availability or stale
count failures. They are not runtime regressions, but they still prevent an
unqualified all-tests-passed claim. git diff --check origin/main...HEAD
cannot represent the uncommitted fixes; it must be rerun after the final commit.

Transformers 5.13 and artifact loading

  • ESM++ small and large exact-checkpoint AutoClass coverage passed for
    AutoConfig, AutoModel, and AutoModelForMaskedLM.
  • ESM++ 6B exact-checkpoint coverage passed all 3 tests in 314 seconds,
    including the sharded checkpoint path.
  • Fresh offline save and reload produced zero missing, unexpected, or mismatched
    weights for the tested AutoClass contracts.
  • Offline FlashAttention 2 and FlashAttention 3 artifact inference passed both
    tests after locked kernel prefetch.
  • The earlier KeyError: 'classifier_dropout' came from the official semantic
    configuration containing that field while the candidate extractor omitted
    it. The shared extractor now compares it on both sides.

ESMC BF16 calibration

The calibration covered all three ESM++ sizes, all five attention backends, a
real biological holdout panel, and generated boundary-length inputs only for
kernel-shape coverage. Of 30 size, backend, and panel combinations, 6 passed
the existing 0.01 relative-L2 target and 24 failed: 23 on relative L2 and one
on relative Q99.9 error. No case exceeded the unchanged 0.03 hard limit.

The predeclared 0.0125 candidate target did not validate across calibration
and independent biological holdouts, so it was rejected and the original
0.01 target was restored. All advertised backends remain present, but this
target failure is still a release blocker. It should not be hidden by silently
weakening the threshold.

ESMFold2 real-protein panel

The panel produced 60 official BF16 bundles, 60 FastPLMs BF16 bundles, and 60
FastPLMs FP8 bundles. All 180 inference jobs completed.

FastPLMs BF16 matched official BF16 in all 60 cases: lDDT-C-alpha was 1.0,
aligned C-alpha RMSD was approximately 1e-6 to 1e-5 angstrom, and all
reported confidence and probability discrepancies were zero.

Variant FP8 hard parity passes FP8 target passes Official/BF16 valid geometry FP8 valid geometry
Standard 11/15 11/15 15/15 15/15
Fast 15/15 15/15 14/15 15/15
Experimental cutoff 2025 9/15 9/15 10/15 10/15
Experimental fast cutoff 2025 13/15 9/15 13/15 13/15
Total 48/60 44/60 52/60 53/60

The complete assertion gate passed 43 of 60 cases. Eight absolute-geometry
failures were reproduced exactly by official BF16 and FastPLMs BF16: fast GFP
seed 17; experimental Crambin seeds 17 and 43 and GFP seeds 17, 29, and 43;
and experimental-fast GFP seeds 29 and 43. These are model or sampling outcomes,
not FastPLMs drift.

Variant FP8/BF16 lDDT-C-alpha FP8/BF16 C-alpha RMSD (angstrom) Probability JSD PAE MAE (angstrom) pLDDT MAE pTM error
Standard 0.407274 to 1.0 0.005415 to 15.848244 0.000007 to 0.039974 0.008672 to 3.619239 0.000248 to 0.073160 0.000152 to 0.189473
Fast 0.970644 to 1.0 0.007825 to 0.472440 0.000008 to 0.000885 0.008564 to 0.199555 0.000197 to 0.005257 0.000039 to 0.004915
Experimental cutoff 2025 0.686798 to 1.0 0.007387 to 3.325241 0.000065 to 0.003790 0.009013 to 0.653813 0.000390 to 0.014975 0.000207 to 0.016251
Experimental fast cutoff 2025 0.759560 to 1.0 0.008734 to 5.394930 0.000045 to 0.009113 0.012215 to 0.683300 0.000509 to 0.021646 0.000047 to 0.016034

ipTM error was zero in every case. The widest FP8 deviations occurred for
Crambin and GFP. The evidence supports explicit FP8 access for informed users,
but it does not support FP8 as the automatic default or as numerically
interchangeable with BF16 across all four variants.

Artifact and aggregate release status

  • Twenty-two artifacts built successfully.
  • The ANKH3 XL artifact did not complete because the remote root filesystem ran
    out of space.
  • Focused ESM++ artifacts were rebuilt and validated after the exact-loading
    fixes.
  • The complete aggregate release suite was not run to a passing conclusion
    because the ESMC target remains unresolved and the artifact build was
    incomplete.

Remaining merge requirements

  1. Resolve the ESMC BF16 backend target without silently weakening the evidence.
  2. Commit the final working tree and rebuild every candidate, reference,
    structure, FP8, and artifact image from that exact commit.
  3. Complete the missing ANKH3 XL artifact build with sufficient disk space.
  4. Run check, artifact, feature, structure, compliance,
    python-matrix, and aggregate release from the exact committed head with
    no required skips.
  5. Rerun git diff --check origin/main...HEAD after the fixes are committed.
  6. Attach the exact-head H100 report and diagnostic artifacts to PR 39.
  7. Obtain at least one approving review.

Until those items are complete, the PR should remain open.

@lhallee

lhallee commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

PR 39 merge-readiness update

Current PR head: 7dbc391 on clean_up.

Implemented changes

  • Fixed causal masked FlashAttention and locked sparse offline kernel loading.
  • Preserved exact tokenizer provenance during artifact builds.
  • Added exact model-state fingerprints to persisted embedding resumes, while skipping expensive hashing when output=None.
  • Rejected schema-v1 embedding resumes and retained the caller-supplied fingerprint escape hatch.
  • Fixed ESM++ Transformers v5 state/config loading and sequence_head.
  • Fixed DPLM2 implicit padding masks and 4-D all-hidden-state embeddings.
  • Fixed output_loading_info=True handling in both ESMFold2 implementations.
  • Made ESMFold2 auto deterministically select BF16.
  • Kept explicit ESMFold2 FP8 opt-in, inference-only, fail-closed, and documented as experimental.
  • Kept ESMC SDPA bit-for-bit exact.
  • Set the ESMC-specific BF16 relative-L2 target to 0.02 for eager, FlexAttention, FlashAttention 2, and FlashAttention 3 while retaining all hard limits and secondary metrics.
  • Added compact biological and boundary-length ESMC calibration fixtures.
  • Updated the release runner to include manifest readiness, exclude experimental FP8 and provisional Boltz2 numerical gates, and record Git, submodule, source, image, GPU, Docker, and environment identities.
  • Fixed canonical ESMFold artifact conversion so only declared unused MLM/contact heads and derived buffers are dropped.
  • Corrected offline artifact validation so native checkpoint heads require exact weight loading while alternate advertised task heads retain load, inference, save, and exact reload coverage.
  • Excluded auto_map packaging provenance from semantic config round-trip comparison.

The 60-case ESMFold2 FP8 parity panel and advisory workflow were not included.

Workstation results

All execution was performed on the designated workstation.

  • Focused review slice: 204 passed
  • Experimental ESMFold2 FP8 coverage: 6 passed
  • Final manifest and ESMFold regressions: 3 passed
  • Artifact generation: 29/29 artifacts built successfully
  • Follow-up artifact-gate representative coverage: 15 passed
  • Fatal Ruff checks: passed
  • git diff --check: passed
  • Local checkout, workstation checkout, and GitHub branch all match 7dbc391
  • Workstation is clean with approximately 265 GB free
  • Named Hugging Face, Torch, XDG, pip, UV, and kernel caches were preserved

Exact-head release status

The aggregate release attempt successfully built all 29 artifacts, then failed in the offline artifact phase because exact checkpoint-key parity was incorrectly being required for alternate task heads. For example, loading an MLM checkpoint through a sequence-classification AutoClass necessarily has a newly initialized classifier and an unused MLM head.

That release-gate bug is fixed in 7dbc391, and representative native-head, alternate-head, save/reload, ESM3 config, ANKH, ESM2 FlashAttention, DPLM, and DPLM2 cases pass.

The complete aggregate suite was not rerun after 7dbc391, so a fully passing exact-head release report is still outstanding. The later native-reference, complete ESMC backend matrix, structure, and Python-matrix phases therefore remain unverified by one final aggregate invocation.

An approving review is also still required before merge.

lhallee added 14 commits July 16, 2026 21:38
@lhallee
Harden release parity and add official goldens
290046a 
Close parity and packaging gaps found during PR review.
Pin compact official golden bundles for every check-tier model.
@lhallee
Expose opt-in ESM++ attention backends
b30ac4e 
Keep Flex Attention and FlashAttention 3 selectable while documenting the
known ESMC-6B BF16 boundary deviations. Track those cases as strict expected
failures so improved kernels force the limitation to be revisited.
@lhallee
Better pooling and tests
82d3402
@lhallee
docs update
3b5a2d5
@lhallee
publish
6e4ffaf
@lhallee
docs update
ceb0fcc
@lhallee
better testing and docs
c240d8a
@lhallee
Update .gitignore
5707e16
@lhallee
getting merge ready
6b37320
@lhallee
Merge branch 'clean_up' of https://github.com/Synthyra/FastPLMs into …
9423b77 
…clean_up
@lhallee
more fixes
ac2df55
@lhallee
u
71a0417
@lhallee
docs and tests update
023f6e9
@lhallee
better actions, docs
dbf6764
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@lhallee